SaFe: a general framework for integrated spatial and feature image search

We present a system for querying for images by the spatial and feature attributes of regions . The system enables the user to nd the images that contain an arrangement of regions similar to that diagrammed in a query image. We propose a general framework which allows for di erent types of features (e.g., color, texture, shape, motion) to be integrated with spatial information in the query process. We demonstrate that integrated spatial and feature querying improves image search capabilities over previous content-based image retrieval methods. INTRODUCTION In this paper, we present the general framework and a prototype system for querying for images by spatial and feature attributes. The spatial and feature (SaFe) system integrates content-based techniques with spatial query methods in order to search for images by arrangements of regions. SaFe has been deployed on-line in an application for querying in a large collection of unconstrained images (more than 650,000 images). Our contribution is the use of fully automated region and feature extraction and indexing, and the integration of spatial and feature image querying. These capabilities of SaFe distinguish it from other recent image retrieval systems (Virage [1], QBIC [2] and Photobook [3]) which do not provide this enhanced functionality. The spatial and feature image query paradigm provides a powerful method for image retrieval. However, it is extremely complex in that it requires that several disparate image query techniques be combined. First, the feature query component requires the assessment of the feature similarities of regions. Second, the spatial query component requires the assessment of the similarities in spatial locations and sizes of regions. Third, the system requires the comparison of images consisting of multiple regions. Last, the system requires that the spatial relationships, such as \above," \below," \near," and so forth, be resolved. As depicted in Figure 1, the overall comparison of images for on-line demo see http://disney.ctr.columbia.edu/safe utilizes both the feature and spatial attributes of the regions in computing their similarity.